Browser Extension TO Removing Dust Using Sequence Alignment and Content Matching

---------------------------------------------------------------------***--------------------------------------------------------------------Abstract If documents of two URLs are similar, then they are called DUST. Similarly, detection of near duplicate documents is complex. The duplicate documents content will be similar but there will be small differences in the content. Different URLs with same content are the source of multiple problems. Most of the existing methods generate very specific rules. So more number of rules are required to increase detection of duplicate URLs. Existing methods cannot detect duplicate url’s across different sites where as candidate rules are derived from URL pairs within the dup-cluster. Existing Methods Complexity is proportional to the number of specific rules generated from all clusters. In the proposed system, the URL normalization process is used which identifies DUST with fetching the content of the URLs. In Proposed system, a new method is present, which obtains a smaller and more general set of normalization rules using multiple sequence alignment. The proposed method is used to generate rules with an acceptable computational cost even when crawling in large scale scenarios. The valid URL’S contents can be fetched from its web.